The factorization method is conventionally known as a method in which an image sequence of a target object is inputted, positions of multiple preset feature points are tracked on the image plane, and a three dimensional shape of the target object is reconstituted from a feature point position sequence obtained by the tracking.
In this factorization method, an observation matrix is generated from the feature point position sequence, and the observation matrix is factorized into a shape matrix showing a shape of the target object (three dimensional positions of the feature points) and a motion matrix showing a motion of the target object.
In the factorization method, to obtain the effective result, all the data forming the observation matrix is needed. Accordingly, it is difficult for the factorization method to be applied to the actual problem that defective data often generates in the feature point position sequence (and data forming the observation matrix) due to hiding of feature points, wrong tracking, and flame-out.
In contrast, the method in which after defective data is removed from the observation matrix, the observation matrix is factorized, and the method in which a partial observation matrix is generated by removing defective data from an observation matrix, and the defective data is estimated from a shape matrix and motion matrix obtained by factorizing the partial observation matrix, and an observation matrix in which the defective data is replaced with the estimation value (for example, see Patent Document 1) is factorized, have been suggested.
However, in the conventional methods, normal data and abnormal data need to be distinguished in generating an observation matrix. Since the distinguishing is difficult, its automatic processing is difficult. Sufficient accuracy of the distinguishing cannot be obtained. Distinguished abnormal data needs to be deleted. The estimation of defective data needs to be repeated. As a result, an amount of necessary calculations disadvantageously increases.
Additionally, in the conventional factorization method, to improve an estimation accuracy of a three-dimensional shape, an image sequence used for calculation needs to be enlarged (namely, the number of dimensions of an observation matrix is increased). Accordingly, an amount of the calculations (calculation time) increases exponentially.
Due to the large amount of the calculations, the conventional factorization method cannot be applied to a computing unit, having a limited calculation ability, such as a three dimensional shape reconstitution device mounted to a vehicle.
On the other hand, there is a known device for estimating a direction of a captured target object in a three-dimensional space in accordance with a three-dimensional shape (hereinafter also called a shape model) of a previously stored human head (hereinafter also called a target object) and multiple feature point sequences extracted from an image of the target object.
In such a device, when a model used for generating a shape model and a captured target object are the same as each other, a direction of the target object can be estimated accurately, but when the model and target object are different from each other, the estimation accuracy disadvantageously decreases.
On the other hand, there is a known method for separately learning shape models (individual models) of target objects to be captured, and for estimating a direction of the target object by use of the learned individual models (for example, see Patent Document 2), and there is a known method for previously producing shape models (average models) showing average shapes of multiple target objects, and for estimating a direction of the target object by use of the average models.
However, disadvantageously, in the method using the individual models, the estimation accuracy is likely to be improved for every target objects, but the estimation of a direction of the target object cannot be started immediately because the learning takes long time, and the estimation accuracy decreases remarkably in case of the failure of the learning.
In the method using the average models, the estimation of a direction of a target object can be started immediately because the learning of the models is unnecessary, but some errors always occur in the estimations, and thus the estimation accuracy cannot be improved because a three-dimensional shape of the target object does not match the three dimensional shapes shown by the average models.    Patent Document 1: JP-2000-113194 A (JP-3711203 B1, corresponding to U.S. Pat. No. 6,628,819)    Patent Document 2: JP-2003-141552A